Data that is stored or transmitted may be protected against storage media failures or other loss by storing extra copies, by storing additional redundant information, or in other ways. One type of redundancy based protection involves using erasure coding. Erasure coding uses additional redundant data to produce erasure codes that protect against ‘erasures’. An erasure code (EC) allows data portions that are lost to be reconstructed from the surviving data. The application of erasure codes to data storage may typically have been for the purpose of recovering data in the face of failures of hardware elements storing the data. Some erasure codes may be simple to compute (e.g., systematic data) while other erasure codes may be more complex to compute (e.g., non-systematic data). Similarly, it may be simpler to recover data using some type of erasure codes (e.g., systematic data) and it may be more complex to recover data using other types of erasure codes (e.g., non-systematic data). However, conventional systems may store erasure codes without considering the complexity and thus the efficiency of recovering data based on the type of erasure code. For example, conventional systems that store data and ECs on disk use sequential disk writes that do not consider the type of EC being written or the different energy requirements for reading/writing data at different zones on a disk.
Adding redundancy introduces overhead that consumes more storage capacity or transmission bandwidth, which in turn adds cost and may increase energy consumption. The overhead added by erasure code processing tends to increase as the protection level provided increases. Ideally, the redundant information may never need to be accessed and thus conventional systems may group all redundancy together and store it in some out of the way place. This one-size-fits-all approach may produce sub-optimal results, particularly concerning energy conservation.
To enhance data protection, different approaches for storing redundant copies of items have been employed. Erasure codes are one such approach. An erasure code is a forward error correction (FEC) code for the binary erasure channel. The FEC facilitates transforming a message of k symbols into a longer message with n symbols so that the original message can be recovered from a subset of the n symbols, k and n being integers, n>k. The symbols may be individual items (e.g., characters, bytes) or groups of items. The original message may be, for example, a file. Optimal erasure codes have the property that any k out of the n code word symbols are sufficient to recover the original message.
Fountain codes are one type of EC. Fountain codes have the property that a potentially limitless sequence of encoding symbols may be generated from a given set of source symbols in a manner that supports ideally recovering the original source symbols from any subset of the encoding symbols having a size equal to or larger than the number of source symbols. A fountain code may be optimal if the original k source symbols can be recovered from any k encoding symbols, k being an integer. Fountain codes may have efficient encoding and decoding algorithms that support recovering the original k source symbols from any k′ of the encoding symbols with high probability, where k′ is just slightly larger than k.
An EC system may be described using an A/B notation, where B describes the total number of encoded symbols that can be produced for an input message and A describes the minimum number of the B encoded symbols that are required to recreate the message for which the encoded symbols were produced. By way of illustration, in a 10 of 16 configuration, or EC 10/16, sixteen encoded symbols could be produced. The 16 encoded symbols could be spread across a number of drives, nodes, or geographic locations. The 16 encoded symbols could even be spread across 16 different locations. In the EC 10/16 example, the original message could be reconstructed from 10 verified fragments. Conventionally, the 16 encoded symbols have been treated equally, which may have produced sub-optimal results, particularly with respect to conserving energy.
Prior Art FIG. 1 illustrates an example set of systematic erasure codes. Prior Art FIG. 1 illustrates an original message 100 that has sixteen symbols S1, S2, . . . S16 (k=16) and that reads “original message”. The symbols S1 . . . S16 may be referred to as the plaintext symbols for message 100. While the symbol size is one character, different symbol sizes may be employed. Message 100 is provided to erasure encoder 110. Erasure encoder 110 uses a generator matrix 120 to produce erasure codes 130. In this example, erasure encoder 110 produces erasure codes EC1, EC2, . . . ECn (n>k). The erasure codes include systematic erasure codes EC1 . . . EC16 (EC1 . . . ECk), which correspond directly to S1 . . . S16 (S1 . . . Sk). In this embodiment, at least EC1 . . . EC16 may be the same size as S1 . . . S16. For example, if the symbols S1 . . . S16 are one byte each, then the symbols EC1 . . . EC16 may also be one byte each. The erasure codes also include EC17 . . . ECn (ECk+1 . . . ECn), which do not correspond directly to any of S1 . . . Sk. In one embodiment, ECk+1 . . . ECn may be parity information. In another embodiment, ECk+1 . . . ECn may be other information that facilitates recreating the original message. In this embodiment, ECk+1 . . . ECn do not need to be the same size as S1 . . . S16. For example, if S1 . . . S16 are one byte, ECk+1 . . . ECn do not have to be one byte. While the systematic erasure codes are illustrated as being generated in order and in locations directly corresponding to the input message, in some embodiments, the systematic erasure codes may be distributed through the ECn codes and may be out of order.
The original message 100 can be recreated from any 16 of the systematic erasure codes EC1 . . . ECn. If EC1 . . . ECk are available, then original message 100 can be recreated without performing erasure code decoding. If any of EC1 . . . ECk are not available, then original message 100 can still be recreated but erasure code decoding would be necessary. Conventionally, all of the erasure codes EC1 . . . ECn may have been treated identically with respect to how they were stored. Conventionally, the generator matrix may have treated the generation of all the erasure codes EC1 . . . ECn identically. This may have produced sub-optimal results with respect to generating erasure codes, storing erasure codes, and recovering data using erasure codes.
Prior Art FIG. 2 illustrates an example set of non-systematic erasure codes. Prior Art FIG. 2 illustrates an original message 200 that also has sixteen symbols S1, S2, . . . S16 (k=16) and that reads “original message”. While the symbol size is one character, different (e.g., larger) symbol sizes are likely to be employed. Message 200 is provided to erasure encoder 210. Erasure encoder 210 uses a generator matrix 220 to produce erasure codes 230. In this example, erasure encoder 210 produces non-systematic erasure codes EC1, EC2, . . . ECn (n>k). EC1, EC2, . . . ECn do not correspond directly to any of S1 . . . S16 as was the case for systematic erasure codes 130 (Prior Art FIG. 1). Instead, EC1, EC2, . . . ECn are the result of processing symbols S1 . . . S16 with the matrix 220 as controlled by erasure encoder 210. EC1, EC2, . . . ECn do not have to be the same size as S1 . . . S16. For example, S1 . . . S16 may be one byte each that represent one ASCII character each while EC1, EC2, . . . ECn are two bytes each. The different erasure codes EC1, EC2, . . . ECn may be suitable for correcting different types of errors. For example, some erasure codes may be suitable or even optimal for correcting for single erasures, other erasure codes may be suitable or even optimal for correcting for two erasures, and yet other erasure codes may be suitable or even optimal for correcting for three or more erasures. Conventionally, the generator matrix 220 may have treated the generation of these erasure codes identically. Additionally, the erasure codes may have been stored without considering the suitability and thus efficiency of recovering data that is experiencing a certain type of erasure.
Prior Art FIG. 3 illustrates a conventional generator matrix 300. Typically, an erasure coder uses a generator matrix like matrix 300 for encoding data. An input message (e.g., plaintext) and the generator matrix 300 may be processed to produce erasure codes. For example, an input message and generator matrix 300 may be multiplied together using matrix multiplication to produce erasure codes. Conventionally, generator matrix 300 did not account for the different types of erasure codes that were produced by the generator matrix 300. For example, rows in the generator matrix 300 that yield systematic data and non-systematic data may have been placed without consideration of the suitability and thus efficiency of using the resulting erasure code to recover data that was suffering from different types of erasure conditions.